Single-kernel models of single-voxel visual selectivities in convolution neural networks

The translation of retinal images into recognizable objects and scenes is not yet well understood. Beyond edgedetection in primary visual cortex, higher stages of cortical representation are still uncertain. We use a multi-layer convolutional neural network (Krizhevsky, 2012) to provide models for visual selectivities in the ventral visual pathway. We examine individual neural units, or ”kernels”, in CNN layer 2, correlating kernel activity to single fMRI voxel activity for 1750 natural images (Kay, 2008). Building on Güçlü (2015), we find most significant voxel-kernel correlations in V2, with additional matches throughout the ventral pathway. Notably, only 25% of kernels correlate with voxel responses — many voxels correlate with a consistent small set of kernels. Inhibition of voxel response for kernel selectivities also was observed. Our results indicate a limited number of CNN kernels may be used to gain a finer understanding of voxel level representations in the mid-level ventral visual pathway.